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ABSTRACT 

Six compensatory education projects, all of which 
were effective, are analyzed. The purpose of the Project Information 
Packages (PIPs) mandated by the Dnited States Office of Education 
(OSOE) was to determine whether compensatory projects could be 
packaged with sufficient clarity and in sufficient detail to 
encourage and enable their replication at sites where educational 
needs had not been adequately met by existing programs of 
instruction. The three stages of replication: selection/adoption, 
start-up, and operation are examined and analyzed. The major problems 
in describing the replication mechanism are explored. These concerned 
the differences between effective projects in the areas of: 
management and instructional features of the projects and in the role 
of the individual teacher in deter mining the instruction of the 
student. (JP) 
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ANALYSIS OF SCHOOL PROJECTS FOR THE DEVELOPMENT OF 
PROJECT INFORJLVnON PACICAGES (PIPs) 

The analysis ol: projects for the development of Project Informa- 
tion Packages contrasts sharply with the analyses described by the 
previous speakers. U^ere the other studies hcive analyzed a range of 
programs looking for features which distinguish effective programs 
from ineffective ones, RMC analy^ied a small number of projects, 
all of which were effective, looking for the unique features which 
led to the success of each project* Since no ineffective projects 
were included in the study, no dimensions could be defined which would 
distinguish projects in terras of effectiveness. In any case, the projects 
in question shared few comraon features at the level of specificity 
required for diffusion to new sites. 

Certainly, a statistical search for comraon features would not have 
helped in this effort to package projects for dissemination. Instead, 
a form of systems analysis was used to identify project eleraents which 
contributed to project effectiveness, and this analysis is described 
belov;. Before turning to the analysis, however, I would like briefly to 
describe the history of the packaging studies in which RMC has been in- 
volved. I would also like to consider two issues which are central to 
current efforts to isolate and comraunicate essential project features: 
(a) Just how effective are exemplary projects in corapensatory education? 
and (b) VJhere should we be looking if ve want to find the things that 
produce apparent achievement gains? 

History of the Packa<^in^ Studies 

TheUSOE has long felt that the effectiveness of federally funded 
programs — particularly those intended to provide compensatory services 



to disadvantaged children — could bo t;i-eatly onhaaceil through the diffu- 
sion of education projects and practices proven to be eff^iccive. It be- 
came clear recently, however, that the simple difJSGT.unat Lon of informa- 
tion about such projects and practices v/as not a sufficient nechanisn 
for bringing about v/idesprcad replication. For this reason, a decision 
was nade in 1973 to attempt a more complete packaging of several exem- 
plary projects. The purpose of this undertaking was to determine whether 
projects could be packaged with sufficient clarity and in sufficient de- 
tail to encourage and enable their replication at sites where educational 
needs liad not been adequately met by existing programs of instruction. 

In June K 3 a twelve-month research and develpment contract was 
awarded to the l^iC Research Corporation by USOI:: to identify up to eight 
compensatory education ^^projects and develop replication packages for 
tliem. Six exemplary projects v^'cre identified, and packages \/c:re de- 
livered to sites for field testing late in the summer of 1974. A de- 
scription of the entire contract effort is presented in a report by 
Tallmadge (1974). 

A two-year evaluation of the PIP field test was the second step 
in answering the question of PIP viability. The field test ron^sisted 
of attempts by two to five sites to replicate each of the six projects, 
a total of 19 replications in all. The evaluation of the field test 
was conducted by SRI with RHC as a subcontractor. The first-year eval- 
uation activities included five visits to each site to observe progress 
and collect student test data. RllC participated in these site visits 
and provided general support but was primarily responsible for develop- 
ing recommendations for PIP revisions (Horst, Piestrup, Foat , & Binkley, 
1975). 

The general conclusion of the field test was that the PIPs were 
fairly successful in enabling accurate replication of the exemplary 
projects (Stearns, 1977). However, the field test also provided many 
new insights into project analysis and PIP design. Based on these in- 
sights, the original projects were visited again and re-analyzed, the 



riFs uc-.ro redesigned, and coniploto ].y new packages ucro ptrod'icud. Tlics«i 
siiCond-*^»enera tion PIPs iwy?. ^.uri'catly boinc^ d icseniaatcd on a aal" lonwide 
basiii. This difnui^iion of tort is jjoing evaluated by Lhe Anerlcan IrKiLf.- 
tuLus I'l^r Resciarc.h, vjith PJ{C as subcontracLor . 

How Effective Are i:>:e:nplary rrojects? 

VJliile the nominal purpose of this series of studies has been Lo ex- 
plore th.e potential of packa-;ing for the diffusion of exeriplary projects, 
the ultimate goal has been to help solve the reading and nath problems oE 
educationally disadvantaged students. In particular we have been asked 
to identify and package tae ''essential elements^' of the exemplary proj- 
ects; that is, those elements v;hich will, if replicated in other sites, 
produce the same achievement: gains obtained at the original project sites. 
Unfortunately there are three interrelated problems which ilriit the abil- 
ity to diffuse project effectiveness. Each of these problems is dis- 
cussed briefly below. 

The basic model is wrong * The fundamental problem has been in con- 
centrating on the impact of the ^'project** while treating other sources 
of impact rather lightly. To belabor the point, we have acted as if 

Gain = f (Project) . 

In fact of course, if the project is replicated in very different sur- 
roundings, with very different students and teachers, gains can be ex- 
pected to differ too. In addition, we have found that the way in which 
achievement is measured has a major effect on apparent gains. In other 
v7ords, if we want to examine project success we must start witli a more 
realistic equation : 

Apparent Gain = f(Project, Context, Measurement). 
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T. v/'ill not attempt to dcveloiJ this Gquation i.u any tier Lous way, 
so procisG definiLions for tae throe independent variables arc. not: too 
important. In g^nc».raJ.> the torn *'proje-ct" iji intended to include only 
those elements ot the project \/hich can be packaged for diffusion, such 
as instructional techniques, management procedures, materials, and so 
on. The "context" is a catch-ail concept including the students and 
their environment, school personnel with their skills and attitudes, and 
all other project features that cannot be exported. "Measurement" in- 
cludes the tests which are used, the evaluation design, and the testing 
procedures. It might also include "test wiseness" and similar sources 
of test score variance which involve stude'il learning but v^hich we would 
not want to include under the h.eading of tlie "project." No one would 
doubt that all three classes of variates affect apparent gain, yet as 
we shall see, their relative importance is often overlooked. 

Project effects are small in relation to our expectations . The 
purpose of compensatory projects is to take students who are not ac- 
quiring adequate skills and teach them tiiose skills. Of course, we all 
profess to be realistic in our expectations as to what is possible, but 
a review of some basic components of test scote variance can be quite 
sobering. Figure 1 illustrates the relative magnitudes of three possible 
componenns of reading scores as measured with a standardized test. The 
first component is the individual differences among students. These dif- 
ferences, as we all know, are very large. The second component is the 
normal expected skill increase over the course of a year for a given stu- 
dent. It includes the effects of (a) the regular school program, (b) 
mturation, and (c) all out-of-school learning. At grade 2, the sum of 
these three effects is small when compared to the differences among stu- 
dents. At grade 6 it is smaller still. The third component is tlie dif- 
ference that an effective project might make. For the purpose of illus- 
tration, a value of one-third of a standard deviation with respect to 
the national norm group is shown in Figure 1. IJhile this value is rather 
arbitrary, in practice we have been able to find very few compensatory 
projects with convincing evidence of even this amount of impact. 
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Unfortunately 5 it seems that the scale of Liidividual diCf cre.nces 
is often used as the frame of reference for Judging project effective- 
ness. We know, for example, that the ninetieth percentile student, 
v;hile not a ^;enius, readrs quite well. The tenth percentile student 
reads so poorly that special help is badly needed. We tend to use 
the fiftieth percentile student as our goal for compensatory projectSj 
perhaps because fiftieth percentile reading skills seem minimally 
adequate, perhaps because \/e feel that no American irliould be below 
average in anything. The problem is, as Figure 1 suggests, that such 
a goal implies the need for truly dramatic changes in learning rates; 
far more dramatic than our experience tells us is possible with avail- 
able instructional techniques. \J'nev\ faced v/ith 'these facts, there is 
a tendency to become d isc our aged and to say in ef f ec t , it we can ' t 
have the noon, why even try? RMC docs not take this pus it ton, buL 
before considering v/ha t we v;ould .ulvocatc, there is a third [) rob. Inn 
to discuss. 

Project ef fect s are small in relation to context aiul measurement 
varianc e. The problem i:j not simply that project effects are small in 
relation to our a.;pi rations. They are also small in relation to the vast 
sea of confound Lag conte::tual and m.easurement effects from which we try 
to separate them. The effects of contextual variates are not \/ell quan- 
tified, but I feel most educators would agree that a project which appears 
effective in middle-class suburbia may not have the same success in an 
inr.er-city settirfg. Similarly, a project which helps students whose main 
problem is limited skill in English may not have as great an impact on 
native speakers of English with nore severe educational deficits, and a 
project which depends on exceptional teachers may not work as well with 
teachers having lesser skills. I would go further, and say that the con- 
text can make or break a project. 

Measurement variance is somewhat better understood. We kno;;, for 
example, that for a project which apparently meets our arbitrary stan- 
dard (one-third of a standard deviation gain due to the project), half 
of that gain could easily be due to regression effect error (Anderson 
& Estes, 1976). The use of grade-equivalent scores may account for a 
comparable spurious gain (Tallmadge L Horst, 1976, p. 74). Dropouts 




from the low end of the r«ir.ge can inflate gains croinendously . For ex- 
ample, if the lo'./er AU% of a group is dropped between the pretest and 
the post test, a one-third standard deviation loss in mean score will 
appear to become one-third standard deviation gain (Floberts, 1977). 
Testing too early or too late can also make large apparent differences 
(Horst, Tallmadge, Wood, 1975, p. 13). In addition, the test which 
is used can be a major factor, since a one-third standard deviation gain 
may represent as few as two or three raw score points. Even changing 
from one standardized test to another can make huge differences in 
scores. The effect of changing to other kinds of tests may be much 
larger. Clearly, then, the match between test items and instructional 
content can be critical (BLanchini, 1976). The general picture is one 
of many variates which might produce roughly equal impacts on apparent 
gain. The project is only one such var-iatc, and not necessarily the 
dominant one. 

Finding the Essential Elements 

Given that project impacts as measured by standardized tests are 
small and elusive, one may ask whether there is any justification for 
pursuing the critical project elements which produce these illusive gains. 
I would say yes, because it is clear to any observer that the best proj- 
ects are better than mediocre ones and far superior to no treatment at 
all. Giving ail students access to the best projects available is still 
a worthwhile goal. V/l^ile we cannot now support this assertion by point- 
ing to impacts on student achievement, I believe that fe\; educators would 
really disagree. Hov;ever, some cautions are "in order. 

Limitations of multivariate statistical techniques . The other 
three studies described in this symposium represent straightforward at- 
tempts to find the elements common to effective projects, or more pre- 
cisely, those elements which distinguish effective projects from less 
effective ones. RflC could not apply such procedures for two reasons. 
First, we had only six projects to analyze in depth, and they were all 
effective. In fact, only effective projects v/ere even nominated for 
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our screening procedures. Thus we had no iiieff ectivo projocts Lo pro- 
vide contrast. Secondly, the si:: projects ue exanincrd siiared little 
in connon in terns of. instructional approaches. Thiis, concentrating 
on common features would have eliminated all iastiuctional features 
from, further coasideration. 

This is not to say that connon features are uninportant. Uur in- 
pressions, based on our intensive experience with these six projects 
and our infernal experience with many others are conpletely consistent 
with the findings of the other studies. There certainly are general 
features which characterize effective projects, but I would classify' 
most of then as management rather than instructional features. Further- 
more, while they are probably among the nose important features to con- 
muni cate to a new adopter site, they do not toll the complete nana^^unent 
story. In particular, they cannot include the unique appro aclies devel- 
oped in exceptionally innovative projects. Statistical techuiciues hav«: 
helped to identify the connon management features of effective project.-^, 
but for purposes of diffusion, v/e must also isolate those features which 
are unique to eacii effective project. 

^[y impression is that, while v^ell inplemented projects tend to 
share certain effective management features, any reputable instructional 
approach can work, if it is implemented v;ell. llov;ever , it would also 
be wrong to conclude that specific instructional features are unimpor- 
tant in ci diffusion effort. It seems obvious that different instruc- 
tional approaches may teach different skills, and that a given approach 
nay be more effective in one context than another, even though our cur- 
rent measurement model is so crude that we cannot say exactly what the 
various approaches accomplish. It is also clear that the insructional 
features of an effective approach \;lll be the first thing that a poten- 
tial adopter \;ill looV: for. The fact that nany different approaches 
are comparably effective does not imply that all approaches will \;ork, 
or that all approaches will be equally attractive to all educators. 
Therefore, if we intend to . analyze projects for the purpose of diffus- 
ing them to other sites, our analysis process must identify the ?lnpor- 
tant instructional features as well as the management features. 

iO 
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Diffusins^ }?,ains vs. diffusin<^ pro iect^i . At thc^. 'beginning of the 
PIP clovelopncnt effort, lUlC uas focusing on a rciLher narrowly defined 
question. The question i/aw this, *'Given that a particular effective 
project is to be diffused, to what extent can a package of printed and 
other materials replace the services of a technical consultant?" Ihis 
is an important question, because cliff us ion via a technical consultant 
is expensive, and the consultant (usually the project director who de- 
veloped the project) has a very limited amount of time. The method of 
project analysis used by RMC was developed in direct response to this 
question, and it focuses on identifying project goals and processes. As 
an aside, I believe that we can now answer the original question rather 
well. In fact, if we combine what has been learned in t\vy PIP studies 
v/ith the results of several concurrent diffu^siou studies, v/e can evtin 
give pretty good ansv;ers tu the more general questions of how to dif- 
fuse projects effectively. 

However, as I mentioned earlier, these wore never the real PIP ques 
tions. The real question was v/hether v;e could package the large achieve 
neat gains reported by a small number of exemplary projects and diffuse 
those gains tliroughout the entire educational system. Analyzing project 
V7ith a view to replicating their gains involves much more than capturing 
their goals and processes. Specifically, it requires careful analyses 
of the project "context** and '/measurement . '* Only where the impacts of 
these variates are fully understood can we be confident that any kind 
of "project" analysis will contribute to producing achievement gains in 
other school districts. 

Diffusing measurement approaches and contexts . Since measurement 
and context seem to be major factors in the apparent effectiveness of 
projects, a brief comment on the possibility and desirability of dif- 
fusing these components is warranted. The issue is not a facetious 
one. Measurement effects are generally exportable and ffill into two 
categories. The first category covers errors in evaluation methodol- 
ogy. It is difficult to prove, but my impression is that a substan- 
tial proportion of reported project impacts are the result of widely 
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accepted but Inappropriate i.walviation practices. These practices are 
easy to package, but we probably do nor. wish to dLffu£;e u.hoin. 

However, the second category of evaluation practices is equally 
easy to diffuse and should probably b2 strongly encouraged. These prac- 
tices include selecting relevant tests, and in general designing evalu- 
ations which capture the e::act learning that Lhe project has produced. 
It is important not to pretend that specific new skills are the same as 
a general improvement in reading or math, but it is equally important 
not to belittle real mpacts on student skills sLnply because those 
skills are not accompanied by big jumps in standardized test scores. 

The concept of diffusing project context may at first .seem absurd, 
since context v;as defined as those features which cannot be packaged 
for export. However, v/hile strictly speaking context cannot be ex- 
ported, it is possible to be sure that there is a good match on criti- 
cal context variates before a new site attempts to adopt a proj^jct. 
The concept that the context in a new site must be compatible with a 
given project has been basic to PIP ^^development from the beginning and 
much of the PIP selection material focuses on critical context varialios. 
Unfortunately, it has been difficult to get anyone to take context con- 
siderations seriously, and projects frequently end up in inappropriate 
settings w:'.th predictable results. 

Analyzing a Project for Diffusion 

Independent of the problems in finding effective projects, if a 
project contributes to student achievement gains, other districts may 
very well wish to adopt it. As I ^^avc noted, both management and in- 
structional features may be included among the important components of 
project success. In fact, one of the major features of the original PIP 
projects has been their longevity. All six have survived for more than 
ten years. Obviously, a project v/hich does not survive does not help 
students, and survival requires much more than demonstrating achievement 
gains. Over the past three years, PJIC has evolved an analysis procedure 
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which has provf^n vory useEui in sorting out the many poals and procedures 
v;hxch make up a successful project. Essentially, it is a systems analy- 
sis which organizes the various initial conditions, processes, and out- 
comes in the project. Since the intent is to let other sites replicate 
the original project, the project is described as a component of a rep- 
lication system. 

Stapes Replication . 

The replication of a project is divided into three stages: (a) se- 
lection/adoption, (b) start-up, and (c) operation. Selection/adoption is 
the process whereby sites select tlie projects that suit their needs and 
apply through appropriate channels for funds. Start-up begins with the 
delivery of a PIP and associated funds to a suitable site, and ends when 
the project is ready for operation. In general, start-up is completed by 
the end of the summer vacation. Operation is the instructional and ad- 
ministrative activity of tlie school year, beginning v;it:h tiie Ejrst con- 
tact between teachers and students for instruct Lou or testing. 

Each of the three stages is furtlier broken down into inputs, pro- 
cesses, and outcomes, and these substages are organized to reflect the 
basic project components of personnel, other resources, and students. In 
this model all three components are receivers of inputs and processes, not 
Initiators. For example, in the personnel section, the project is de- 
scribed in terms of what is done _to teachers, not by them. The resulting 
descriptive model is summarized in Figure 2. 

The selection/adoption stage is included in Figure 2 for completeness, 
but it is incidental to the analysis of the project, and will not be dis- 
cussed further here. Hov/ever, several other features oi the replication 
model should be discussed. The intended outcome of the selec: t ion/adopt ion 
stage is a site that meets the requirements for replicating tl;e project 
chosen. This outcome is described in terms of: the characteristics of 
personnel , other resources, and students at the site, and constitutes 
a major component of the inputs to the start-up stage. The reuainder of 
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inputs to start-up include the PI?, inoney, technical assistance, and other 
inputs that are supplied to the site to help in replicating the desired 
project* In other words, assistance supplied to the site is a subset of 
the inputs to the start-up stage. Inputs to the start-up stafi;u also in- 
clude the site itself v/ith its personnel, other resources, and students. 

Similarly, the outcones of start-up are inputs to the operation starve 
However, unlJke start-up, there are very few additional inputs durin[^ op- 
eration. This is a consequence of a basic PIP replication concept: the 
concept that a packa^^e of information (and associated funds) can be suffi- 
cient to produce the replication of a project if delivered to an appropri- 
ate site . To the extent that this concept proved unworkable in the PIP 
field test, new inputs in the forms of monitoring, technical assistance^ 
and the like have been identified for the operation .sta;/,e. 

The process column under each stage includes the activities c«irried 
out at that stage and the timing considerations for each activity. Timing 
includes the amount of tine allotted for each 'ac t iv i ty , the ;;e<;uenco of 
activities, and the period in the school year during', which the activity 
is expected to take place. 

Descriptive Conventions in the Mod el 

The lower section of Figure 2 displays the specific descriptive con- 
ventions (see Horst et al., 1975, Appendix D, for definitions) adopted by 
RlIC in categorizing field test data. Personnel are divided into (a) the 
project director, (b) other project staff, and (c) nonproject personnel. 
These categories were adopted because of the qualitatively different ways 
in which the PIP mechanism interacts v;ith then. The project directors are 
central to the project replication. They are intended to be selected by 
::he tine of the site proposal or grant request, and most of th».» PIP m;ite- 
rials are directed to them. V/liile the PCPs provide them witli some job 
aids in the form of calendars, sample forms, and the like, it is assumed 
that they possess most of the required characteristics at the tine they 
are selected, and much of their PIP m terial simply describes what they 
should accomplish, with minimal suggestions on how to proceed. 
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Personnel inputs and personnel outcones are described in iierns of the 
titles of the personnel required, and their required characteristics. 
Characteristics of pecfionnel, or outcones, are categorized under s kills , 
attitudes , and quantity . In addition, for the purposes of the replication 
niodol there are critical roles, interrelationships, job positions, and the 
like^ which are not considered to be either skills or attitudes. Thus, 
for example, the amount of authority given to a project director by the 
district administration may be one of the most important factors to de- 
scribe. A third category of r oles is provided to encompass ,such personnel 
descriptors. 

Processes related to personnel aru broken down into s election and 
training . Training is taken in the broad sense to include any activitos 
designed to change skills or attitudes. In addition to conventional skill 
training, this definition encompasses all orientation and instr\icL Lonal 
activities. Procedures designed to create roles or authority relatloti- 
ship.s may not fall under either selection or training (e.g., giving a proj- 
ect director the task of hiring te.ichors in order t<) establish his or hor 
authority over them). Such procedures are included luidei; a third category, 
establishing roles . 

The remaining column is labeled "(new) inputs." During start-up, 
these inputs include the PIP, other technical assistance, money, con- 
straints, and, for the original six PIPs, field-test events. Host of 
these inputs should be generally self-explanatory and no special de- 
scriptive conventions v/ere adopted. The exception is the contunt oC the 
PIP, for which the fo]lowiitg conventions are used: 

Information includes descriptions of tasks and activities in terms 
of intended outcomes and their sequences. The significance oE this cate- 
gory for the development of PIPs Ls the assumption that project directors 
and other staff will know how to accomplish the particular tasks and ac- 
tivities described in this brief manner. 




Self-trainln;:^ raaterial s are in tended to help personnel acquire new 
skillfj, and attitudes, and range from iuLornal tips and suggestions to, 
in one ?IP, a programmed tape/slide crainiui^ sequence. Uliiio it is not 
always possible to categorize materials unambiguously under iaf ormat ion 
or training > the distinction is helpful in developing a package because 
it nakes explicit the assumptions on the cixtent to which various person- 
nel will come to the project with the necessary skills or v;ill require 
PIP assistance in acquiring them. By and large, the PIPs were designed 
as information packages with only limited self- training materials. The 
major project training activity, training of project staff, was generally 
assigned to the vev project directors, and it was assumed that they will 
have most of the basic skills needed to conduct such trainin;^. 

Tlie major remaining component of a project is its students. Students 
are described undor the same conventions as personnel. Their characteris- 
tics (outcomes) include skills, attitudes, and quantity; s;al ecc ion and 
training are the processes in which they are involved. 

Host of the description of students and the processes that affect them 
are included under operation processes and outcomes, \7hilo student out- 
comes in terms of skills and attitudes constitute the ultimate goals of any 
educational project, the emphasis of the PIP replication mechanism is on 
the replication of selected instructional processes. These processes are 
included under the operation process of student training . It should be 
noted that these instructional processes can be defined, in large part, in 
terms of teacher behaviors and interactions with materials. Thus, student 
training will encompass the operation outcomes for personnel and other re- 
sources. 

Applying, the Ifodcl 

Conceptually, the model in Figure 2 provides a simple framework i:or 
isolating the critical elements of the PIP replication meclianism. In 
practice, however, there are major problems in describing the replication 
mechanism in terms of this or any other model. In particular, the number 
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of speclEic items that could be listed under each heading is virtually 
infinite, so there is no practicable vay of describing the niechanisTH ex- 
hauFcively. For example, consider the problem of describing the i^kills 
and attitudes of the project director which are assumed to exist at the 
outcome of the selection/adoption stage. It is clear that the project 
director must have appropriate technical and managerial skills and must 
have a gerierally positive attitude tov>;ard the project. However, enumer- 
ating all cf these skills and attitudes at a detailed level is clearly 
impractical, as it would be necessary in the process to exclude all the 
possible human characteritics the project director must not have. At- 
tempting to list all the processes intended to modify personnel charac- 
teristics presents a similar problem. 

Karly attempts by RIIC to produce appropriate lists led to considerable 
frustration. The lists quickly became long and unwieldy with details that 
v;ere obvious or trivial, yet ^;hen used as a basis of comparison for the 
field-test sites, the lists never seemed to have a place for tho. critical 
site-specific problems. I^nat \/as needed was a list of categories to sys- 
teraatize the description; this v/ould provide an overall persj)octive on llie 
mechanism in question and would highlight those parts of the mechanism re- 
quiring revision. Clearly, a systematic procedure for generating such a 
description was needed. 

The procedure which was adopted took as its primary input the field- 
test data on problems encountered by sites. The rationale for this proce- 
dure was sinply that any aspect of the replication process that did not go 
as intended at one or more sites indicated a possible defect in the in- 
tended replication mechanism and a potential point for a recommended re- 
vision. 

The data usgu to develop the model came from a variety of sources. 
The reports of site observations by SRI, PJlC, and USOE site visitors v;ere 
the major source of data. I/liile it V7as not practicable to list every 
sense in which sites were replicating^ successfully, it was quite possible 
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to list the major ways in v/hicli sites are deviatin^^ from intended practices. 
Included v/ere examples of observed failures to replicate, problems observed, 
user comments on reasons for problems, reported ambiguities in the PIPs, 
and lack of information in the PI?s. Observers obtained verbal reports, re- 
viewed marginal notes written in the PIPs by project staff, and looked for 
specific evidence of use or nonuse of PIP components. In addition, all re- 
quests by project staff for assistance from outside sources were monitored 
via contact report forms. 

The first step in the analysis procedure was to screen all of the data 
described above. All reports and other data sources were systematically 
reviewed, each problem or deviation was entered on a separate card, and the 
cards were sorted into the various categories shown in Table 1. Initially, 
cards from each site v/ere processed separately so that 19 different sets 
of cards resulted. 



The resulting picture of the problems at a given site was incomplete. 
V/hile the outcomes of start-up v;ero generally portrayed fairly systemati- 
cally, there were gaps in the preceding columns. The practical problems 
faced by the site observer virtually preclude recording every possible as- 
pect of each process and every possible characteristic of each person in- 
volved in the project and, in many cases, the critical things to look for 
became apparent only after an associated outcome went awry. To identify 
and fill in the gaps, RMC used the following procedure: the start-up out- 
come problems for a given site were organized into manageable units. I'or 
example, one unit might consist of required teacher skills which were not 
present at the end of start-up. The intended mechanisms which v;ere de- 
signed to produce the skills were then systematically reviev>^ed beginning 
with other start-up outcomes, then working backward through the related 
start-up processes, start-up inputs, and selection/adoption outcomes. 

To be specific, when looking for the reasons why teachers lacked cer- 
tain desired attitudes, the other start-up outcomes to consider would 
include, at least, the expected attitudes of the project director toward 
the project and the availability and adequacy of the materials and facili- 
ties. Next, start-up processes would be reviewed to see where breakdo^^s 
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occurred Ln trair.ing the teachers or ordering; nacerials. Tlicin start-up 
Inputs would bo exanined to doterninri v;here the PIP appeared deficient., 
where the outcome;, of selection/adoption were not as expected, and v/hert; 
money, t/SOE con.^tra Lnts , local policies, field-test inputs, and the like, 
affected training or ordering processes- 

As n.i catef,ory of the mechanism was considered, there were thre.e 
judgments to be made: (a) the catep,ory does not bear on the problem under 
consideration (e.^f^,, lack of desired attitudes), either directly or through 
any relation to an intervening category that relates to tht; problem; (b) 
the category is relevant to the proMem, but cards are already prepared for 
the deviatonF. at this site that fall in this category and affect the prob- 
lem; (c) there appear to be relevant deviations in the category that were 
either overlooked in screening tiie data or i\nt recorded by site visitors, 
l/here a deviation v/as simply overlooked, a card was prepared and included 
in the appropriate category. Where thu information was missing (e.g., 
where tltere was no information as to 'why specific materials ^lerc unavail- 
able), a note v/as made to obtain the Information by phone or during a sub- 
sequent site visit. An example of the resulting site description js si)^j\/n 
in Table 1, abridged somewhat for Illustrative purposes. 

Once this backv;ard- ( f rom "outcome" to "input") review proi • .^s was com- 
plete for a given problem at a given site, it was repeated for the ne-:t 
start-up outcome and so on until all of the unintended outcomes were cov- 
ered. As may be observed from Table 1, beginning with teacher attitude 
problems immediately brings in a variety of other outcome problems. Thus, 
successive outcome problems can be processed quickly, since many of their 
causes will have been previously noted. 

Before the process was considered finished for the site, a "forward" 
review was also completed. This involved taking each deviation in the se- 
lection/adoption outcomes and looking for expected problems at successive 
stages. This review was carried out in a manner analogous to that of the 
backward review, with each category of each stage given individual consid- 
eration. The result of this entire procedure was a set of cards for each 
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site, catac;orized accordinf^ to tilie rr.oael ia Fi<;',are 2, and dc:Jtailiiig for 
each siLe the critical stGr)S in th'.^ replication riechanisn \;here problens 
occur rt?d. An exanplo. of the contc-nt for one site (start-up personnel pro- 
cesses) is presented in llorst» et al. (1975a). 

It should be rcenphasised that cho description uf individual sites 
v/as in terns of the actual problems they oncountored. The next step in 
processing the field-test data was to combine the data from all sites that 
used a given PIP and to restate the probltims poi^itively in terns of the 
processes and characteristics that make up the intended replication mech- 
anism. The products of this step were six descriptions, one for each PIP, 
of the project replication mechanism. These dL'scriptions are, ia essence, 
similar to those developed by SRI/RIIC from the PIPs as the firs^t step in 
the study. The two major differences are that the sta^^jes of the replica- 
tion mechanism are explicitly modeled, and the steps whicli h.we proven 
critical in the field test are systematically included in the descriptions. 
These descriptions provided the required basis for speclCyint; r(.*qulrcd PIP 
revisions. 

For the purposes of describing the overall findings relevant to IMP 
revision and ensuring that formats of PI? descriptions are consistent 
across PIPs, one final processing step was undertaken. The descriptions 
for the six individual PIPs were combined and summarized into a general- 
ized model of the PIP replication mechanism. Basically this model was an 
elaboration of the model sho^vn in Figure 2, but, like the individual PIP 
descriptions, the model reflects the experience of the field test. 

Generalizations from the Analysis 

V.^ile the analysis described above did not produce a set of features 
common to effective projects, some generalizations about thi- dif f erences 
between effective projects can be made. One, which has buuii r^Mitioued 
above, relates to the distinction between management swd instructional 
features of a pioject. Certain m.anagement features (cc;., naialaining 
good relations with non-project personnel) are shared by many effective 



projects as the othi^r speakers have noted. Others vary v;Ldely (for ex- 
ample, the degree and techniques of control exercised by the project 
director), flany of the nianngenent techniques are focused on projcict 
survival and have little direct influence on the instruction received 
by the students. 

A second dinension on which effective projects differ is the role 
of the individual teacher in deternining the instruction of the student. 
At one extreme night be projects using conputer-control led terminals in 
which the only "teacher" is a monitor who maintains order. At Llie other 
extreme, some projects simply hire excellent teachers and provide? tiltem 
with the resources they need tu teach as they choose. In bc»LWL*en fall 
a complete range of pro jects including-, programed \ ■ in:; pro jects in 
v/hich paraprof ess ^ .ollow explicitly proscrLbc^d Lutorin;?. sequences, 
and te;-.'oae •-centered projects in v/h5c:h aver.jge teachers receivii frequent 
train in^^ and monitoring in project-sj-^ecif led techniques. 

A final difference of critical importance to a polenlial adopter site 
concerns the interaction betv;een nanagenent and i ns t rue t ion<i L approaches. 
For example, it is obvious that a laissez f aire , teacher-centered managu- 
ment style cannot work with paraprof essional staff; it is equally true 
that a project based on tight control of highly skilled specialists may 
fail to work. In short, when attempting to analy;:e an effective project 
for the purpose of diffusion, it must be considered as a unique entity, 
and the complex interrelationships among many project elements must be 
described in terms of their contribution to the achievement of the ulti- 
mate project goals. 



1. Tne impact of "projects" is small in relation to our aspirations for 
these projects. 

2v The impact of "projects" is small in relation to the many sources of 
confounding var iance, 

3. The best compensatory projects are much better than average projects, 
and far superior to no treatment at all. However in order to measure 
their superiority we need better, criteria and measures of project 
success . 

4. The diffusion of apparent achievement gains v;ould require careful 
analysis of project measurement techniques and context, as well as 
analysis of the management and instructional features. Me can expect 
disnppointing results from suc'a an effort. 

5. Multivariate techniques have provided information \/hich the project 
packager should know, but quite different analyses are needed to 
identify the features that contribute to the success of an individual 
project. 
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